{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This is implementation of the Recommender training\n",
    "\n",
    "This implementation takes a list of users and their purchasing history to calculate prediction\n",
    "on the probability that they would by a certain product.\n",
    "The implementation is structured in 2 parts:\n",
    "1. Build rating matrix based on the purchasing history. The implementation is based on this blog post\n",
    "https://medium.com/datadriveninvestor/how-to-build-a-recommendation-system-for-purchase-data-step-by-step-d6d7a78800b6\n",
    "2. Build collabarative filtering model based on the rating matrix. The implementation is based on this project https://github.com/Piyushdharkar/Collaborative-Filtering-Using-Keras \n",
    "\n",
    "Implementation is leveraging Minio for storing both source data and result models\n",
    "\n",
    "It also uses Python kubernetes client for re starting model server pod\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Install libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-1.0.1-cp36-cp36m-manylinux1_x86_64.whl (10.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 10.1 MB 3.2 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.18.1)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas) (2.8.1)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas) (2019.3)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.6.1->pandas) (1.11.0)\n",
      "Installing collected packages: pandas\n",
      "Successfully installed pandas-1.0.1\n",
      "Collecting keras\n",
      "  Downloading Keras-2.3.1-py2.py3-none-any.whl (377 kB)\n",
      "\u001b[K     |████████████████████████████████| 377 kB 3.2 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: h5py in /usr/local/lib/python3.6/dist-packages (from keras) (2.10.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras) (1.18.1)\n",
      "Requirement already satisfied, skipping upgrade: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from keras) (1.0.8)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from keras) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.9.0 in /usr/lib/python3/dist-packages (from keras) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras) (5.3)\n",
      "Installing collected packages: keras\n",
      "Successfully installed keras-2.3.1\n",
      "Collecting minio\n",
      "  Downloading minio-5.0.7-py2.py3-none-any.whl (71 kB)\n",
      "\u001b[K     |████████████████████████████████| 71 kB 1.9 MB/s eta 0:00:011\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: python-dateutil in /usr/local/lib/python3.6/dist-packages (from minio) (2.8.1)\n",
      "Requirement already satisfied, skipping upgrade: urllib3 in ./.local/lib/python3.6/site-packages (from minio) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.6/dist-packages (from minio) (2019.3)\n",
      "Requirement already satisfied, skipping upgrade: certifi in /usr/local/lib/python3.6/dist-packages (from minio) (2019.11.28)\n",
      "Collecting configparser\n",
      "  Downloading configparser-4.0.2-py2.py3-none-any.whl (22 kB)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->minio) (1.11.0)\n",
      "Installing collected packages: configparser, minio\n",
      "Successfully installed configparser-4.0.2 minio-5.0.7\n",
      "Collecting kubernetes\n",
      "  Downloading kubernetes-10.0.1-py2.py3-none-any.whl (1.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.5 MB 3.4 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: certifi>=14.05.14 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (2019.11.28)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.3 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (2.8.1)\n",
      "Requirement already satisfied, skipping upgrade: pyyaml>=3.12 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (5.3)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.9.0 in /usr/lib/python3/dist-packages (from kubernetes) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (0.57.0)\n",
      "Requirement already satisfied, skipping upgrade: urllib3>=1.24.2 in ./.local/lib/python3.6/site-packages (from kubernetes) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: setuptools>=21.0.0 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (45.1.0)\n",
      "Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from kubernetes) (2.22.0)\n",
      "Requirement already satisfied, skipping upgrade: google-auth>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from kubernetes) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: requests-oauthlib in /usr/local/lib/python3.6/dist-packages (from kubernetes) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/lib/python3/dist-packages (from requests->kubernetes) (2.6)\n",
      "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->kubernetes) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth>=1.0.1->kubernetes) (0.2.8)\n",
      "Requirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth>=1.0.1->kubernetes) (4.0)\n",
      "Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth>=1.0.1->kubernetes) (4.0.0)\n",
      "Requirement already satisfied, skipping upgrade: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib->kubernetes) (3.1.0)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes) (0.4.8)\n",
      "\u001b[31mERROR: kfp 0.2.2.1 has requirement kubernetes<=10.0.0,>=8.0.0, but you'll have kubernetes 10.0.1 which is incompatible.\u001b[0m\n",
      "Installing collected packages: kubernetes\n",
      "  Attempting uninstall: kubernetes\n",
      "    Found existing installation: kubernetes 10.0.0\n",
      "    Uninstalling kubernetes-10.0.0:\n",
      "      Successfully uninstalled kubernetes-10.0.0\n",
      "Successfully installed kubernetes-10.0.1\n",
      "Collecting kfmd\n",
      "  Downloading kfmd-0.1.8.tar.gz (29 kB)\n",
      "Building wheels for collected packages: kfmd\n",
      "  Building wheel for kfmd (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfmd: filename=kfmd-0.1.8-py3-none-any.whl size=65919 sha256=c65ab8ff649134dbe6c8391743d5361546e5b29e6df9c0ff13915c99b67be1e7\n",
      "  Stored in directory: /home/jovyan/.cache/pip/wheels/54/6b/5c/f063f501d5c632c93566ed967f2f0c36bad3b384d68c83aa65\n",
      "Successfully built kfmd\n",
      "Installing collected packages: kfmd\n",
      "Successfully installed kfmd-0.1.8\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas --upgrade --user\n",
    "!pip install keras --upgrade --user\n",
    "!pip install minio --upgrade --user\n",
    "!pip install kubernetes --upgrade --user\n",
    "!pip install kfmd --upgrade --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "from minio import Minio\n",
    "from keras.models import Model\n",
    "from keras.layers import *\n",
    "from keras.losses import *\n",
    "import tensorflow as tf\n",
    "import os\n",
    "from kfmd import metadata\n",
    "from datetime import datetime\n",
    "from keras import backend as K\n",
    "from kubernetes import client as k8s_client, config as k8s_config\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a workspace, run and execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "execTime = datetime.utcnow().isoformat(\"T\")\n",
    "ws = metadata.Workspace(\n",
    "    # Connect to metadata-service in namesapce kubeflow in k8s cluster.\n",
    "    backend_url_prefix=\"metadata-service.kubeflow.svc.cluster.local:8080\",\n",
    "    name=\"recommender\",\n",
    "    description=\"a workspace for saving recommender experiments\")\n",
    "r = metadata.Run(\n",
    "    workspace=ws,\n",
    "    name=\"run-\" + execTime ,\n",
    "    description=\"recommender run\",\n",
    ")\n",
    "exec = metadata.Execution(\n",
    "    name = \"execution\" + execTime ,\n",
    "    workspace=ws,\n",
    "    run=r,\n",
    "    description=\"recommender ML execution\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Read data\n",
    "\n",
    "For reading data we are using two diffierent approaches:\n",
    "1. We use Tensorflow build in support to write resulting model to Minio\n",
    "2. We use Minio APIs to read source data using Pandas. We could of use Boto APIs here instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minio parameters : URL  minio-service.kubeflow.svc.cluster.local:9000  key  minio  secret  minio123\n"
     ]
    }
   ],
   "source": [
    "minio_endpoint = os.environ.get('MINIO_URL', 'minio-service.kubeflow.svc.cluster.local:9000')\n",
    "minio_key = os.environ.get('MINIO_KEY', 'minio')\n",
    "minio_secret = os.environ.get('MINIO_SECRET', 'minio123')\n",
    "\n",
    "print('Minio parameters : URL ', minio_endpoint, ' key ', minio_key, ' secret ', minio_secret)\n",
    "\n",
    "os.environ['AWS_ACCESS_KEY_ID'] = minio_key\n",
    "os.environ['AWS_SECRET_ACCESS_KEY'] = minio_secret\n",
    "os.environ['AWS_REGION'] = 'us-west-1'\n",
    "os.environ['S3_REGION'] = 'us-west-1'\n",
    "os.environ['S3_ENDPOINT'] = minio_endpoint\n",
    "os.environ['S3_USE_HTTPS'] = '0'\n",
    "os.environ['S3_VERIFY_SSL'] = '0'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "minioClient = Minio(minio_endpoint,\n",
    "                    access_key=minio_key,\n",
    "                    secret_key=minio_secret,\n",
    "                    secure=False)\n",
    "\n",
    "minioClient.fget_object('data', 'recommender/users.csv', '/tmp/users.csv')\n",
    "customers = pd.read_csv('/tmp/users.csv')\n",
    "minioClient.fget_object('data', 'recommender/transactions.csv', '/tmp/transactions.csv')\n",
    "transactions = pd.read_csv('/tmp/transactions.csv')\n",
    "\n",
    "#Log experiment data set\n",
    "data_set = exec.log_input(\n",
    "        metadata.DataSet(\n",
    "            description=\"recommender current transactions and customers\",\n",
    "            name=\"Current transactions and customers\",\n",
    "            version=execTime,\n",
    "            uri=\"minio:/tmp/transactions.csv; minio:/tmp/users.csv\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27773</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerId\n",
       "0        1553\n",
       "1       20400\n",
       "2       19750\n",
       "3        6334\n",
       "4       27773"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(customers.shape)\n",
    "customers.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(62483, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "      <th>products</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2|2|23|68|68|111|29|86|107|152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>111|107|29|11|11|11|33|23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>164|227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2|2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerId                        products\n",
       "0           0                              20\n",
       "1           1  2|2|23|68|68|111|29|86|107|152\n",
       "2           2       111|107|29|11|11|11|33|23\n",
       "3           3                         164|227\n",
       "4           5                             2|2"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(transactions.shape)\n",
    "transactions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Data preparation\n",
    "\n",
    "Our goal here is to break down each list of items in the products column into rows \n",
    "and count the number of products bought by a user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>111.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>107.0</td>\n",
       "      <td>152.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerId     0    1     2     3     4      5     6     7      8      9\n",
       "0           0  20.0  NaN   NaN   NaN   NaN    NaN   NaN   NaN    NaN    NaN\n",
       "1           1   2.0  2.0  23.0  68.0  68.0  111.0  29.0  86.0  107.0  152.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1: split product items\n",
    "transactions['products'] = transactions['products'].apply(lambda x: [int(i) for i in x.split('|')])\n",
    "transactions.head(2).set_index('customerId')['products'].apply(pd.Series).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "      <th>productId</th>\n",
       "      <th>purchase_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>23.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>29.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>68.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>86.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>107.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>111.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>152.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerId  productId  purchase_count\n",
       "0           0       20.0               1\n",
       "1           1        2.0               2\n",
       "2           1       23.0               1\n",
       "3           1       29.0               1\n",
       "4           1       68.0               2\n",
       "5           1       86.0               1\n",
       "6           1      107.0               1\n",
       "7           1      111.0               1\n",
       "8           1      152.0               1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2: organize a given table into a dataframe with customerId, single productId, and purchase count\n",
    "pd.melt(transactions.head(2).set_index('customerId')['products'].apply(pd.Series).reset_index(), \n",
    "             id_vars=['customerId'],\n",
    "             value_name='products') \\\n",
    "    .dropna().drop(['variable'], axis=1) \\\n",
    "    .groupby(['customerId', 'products']) \\\n",
    "    .agg({'products': 'count'}) \\\n",
    "    .rename(columns={'products': 'purchase_count'}) \\\n",
    "    .reset_index() \\\n",
    "    .rename(columns={'products': 'productId'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Create data with user, item, and target field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(133585, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "      <th>productId</th>\n",
       "      <th>purchase_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerId  productId  purchase_count\n",
       "0           0          1               2\n",
       "1           0         13               1\n",
       "2           0         19               3\n",
       "3           0         20               1\n",
       "4           0         31               2"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.melt(transactions.set_index('customerId')['products'].apply(pd.Series).reset_index(), \n",
    "             id_vars=['customerId'],\n",
    "             value_name='products') \\\n",
    "    .dropna().drop(['variable'], axis=1) \\\n",
    "    .groupby(['customerId', 'products']) \\\n",
    "    .agg({'products': 'count'}) \\\n",
    "    .rename(columns={'products': 'purchase_count'}) \\\n",
    "    .reset_index() \\\n",
    "    .rename(columns={'products': 'productId'})\n",
    "data['productId'] = data['productId'].astype(np.int64)\n",
    "\n",
    "print(data.shape)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Normalize item values across users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>productId</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "productId   0    1    2    3    4    5    6    7    8    9    ...  290  291  \\\n",
       "customerId                                                    ...             \n",
       "0           NaN  2.0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN  NaN   \n",
       "1           NaN  NaN  6.0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN  NaN   \n",
       "2           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN  NaN   \n",
       "3           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN  NaN   \n",
       "4           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN  NaN   \n",
       "\n",
       "productId   292  293  294  295  296  297  298  299  \n",
       "customerId                                          \n",
       "0           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "1           NaN  1.0  NaN  NaN  1.0  NaN  NaN  NaN  \n",
       "2           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "3           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "4           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "\n",
       "[5 rows x 300 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_matrix = pd.pivot_table(data, values='purchase_count', index='customerId', columns='productId')\n",
    "df_matrix.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(24429, 300)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>productId</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>290</th>\n",
       "      <th>291</th>\n",
       "      <th>292</th>\n",
       "      <th>293</th>\n",
       "      <th>294</th>\n",
       "      <th>295</th>\n",
       "      <th>296</th>\n",
       "      <th>297</th>\n",
       "      <th>298</th>\n",
       "      <th>299</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 300 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "productId   0    1         2    3    4    5    6    7    8    9    ...  290  \\\n",
       "customerId                                                         ...        \n",
       "0           NaN  0.1       NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN   \n",
       "1           NaN  NaN  0.166667  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN   \n",
       "2           NaN  NaN       NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN   \n",
       "3           NaN  NaN       NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN   \n",
       "4           NaN  NaN       NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  ...  NaN   \n",
       "\n",
       "productId   291  292  293  294  295  296  297  298  299  \n",
       "customerId                                               \n",
       "0           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "1           NaN  NaN  0.0  NaN  NaN  0.0  NaN  NaN  NaN  \n",
       "2           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "3           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "4           NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  \n",
       "\n",
       "[5 rows x 300 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_matrix_norm = (df_matrix-df_matrix.min())/(df_matrix.max()-df_matrix.min())\n",
    "print(df_matrix_norm.shape)\n",
    "df_matrix_norm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(133585, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerId</th>\n",
       "      <th>productId</th>\n",
       "      <th>scaled_purchase_freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0.133333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    customerId productId  scaled_purchase_freq\n",
       "9            9         0              0.133333\n",
       "25          25         0              0.133333\n",
       "32          33         0              0.133333\n",
       "35          36         0              0.133333\n",
       "43          44         0              0.133333"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a table for input to the modeling\n",
    "\n",
    "d = df_matrix_norm.reset_index()\n",
    "d.index.names = ['scaled_purchase_freq']\n",
    "data_norm = pd.melt(d, id_vars=['customerId'], value_name='scaled_purchase_freq').dropna()\n",
    "print(data_norm.shape)\n",
    "data_norm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 Preparing data for learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28606\n",
      "300\n",
      "[    9    25    33 ... 26873 26998 28066]\n",
      "[  0   0   0 ... 299 299 299]\n",
      "[0.13333333 0.13333333 0.13333333 ... 0.         0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "customer_idxs = np.array(data_norm.customerId, dtype = np.int)\n",
    "product_idxs = np.array(data_norm.productId, dtype = np.int)\n",
    "\n",
    "ratings = np.array(data_norm.scaled_purchase_freq)\n",
    "\n",
    "n_customers = int(data_norm['customerId'].drop_duplicates().max()) + 1\n",
    "n_products = int(data_norm['productId'].drop_duplicates().max()) + 1\n",
    "n_factors = 50\n",
    "\n",
    "input_shape = (1,)\n",
    "\n",
    "print(n_customers)\n",
    "print(n_products)\n",
    "print(customer_idxs)\n",
    "print(product_idxs)\n",
    "print(ratings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Tensorflow Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create TF session and set it in Keras\n",
    "sess = tf.Session()\n",
    "K.set_session(sess)\n",
    "K.set_learning_phase(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Model Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DeepCollaborativeFiltering(Model):\n",
    "    def __init__(self, n_customers, n_products, n_factors, p_dropout = 0.2):\n",
    "        x1 = Input(shape = (1,), name=\"user\")\n",
    "\n",
    "        P = Embedding(n_customers, n_factors, input_length = 1)(x1)\n",
    "        P = Reshape((n_factors,))(P)\n",
    "\n",
    "        x2 = Input(shape = (1,), name=\"product\")\n",
    "\n",
    "        Q = Embedding(n_products, n_factors, input_length = 1)(x2)\n",
    "        Q = Reshape((n_factors,))(Q)\n",
    "\n",
    "        x = concatenate([P, Q], axis=1)\n",
    "        x = Dropout(p_dropout)(x)\n",
    "\n",
    "        x = Dense(n_factors)(x)\n",
    "        x = Activation('relu')(x)\n",
    "        x = Dropout(p_dropout)(x)\n",
    "\n",
    "        output = Dense(1)(x)       \n",
    "        \n",
    "        super(DeepCollaborativeFiltering, self).__init__([x1, x2], output)\n",
    "    \n",
    "    def rate(self, customer_idxs, product_idxs):\n",
    "        if (type(customer_idxs) == int and type(product_idxs) == int):\n",
    "            return self.predict([np.array(customer_idxs).reshape((1,)), np.array(product_idxs).reshape((1,))])\n",
    "        \n",
    "        if (type(customer_idxs) == str and type(product_idxs) == str):\n",
    "            return self.predict([np.array(customerMapping[customer_idxs]).reshape((1,)), np.array(productMapping[product_idxs]).reshape((1,))])\n",
    "        \n",
    "        return self.predict([\n",
    "            np.array([customerMapping[customer_idx] for customer_idx in customer_idxs]), \n",
    "            np.array([productMapping[product_idx] for product_idx in product_idxs])\n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64\n",
    "val_per = 0.25\n",
    "epochs = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 Model Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "Model: \"deepcollaborativefiltering_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "user (InputLayer)               (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "product (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 1, 50)        1430300     user[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 1, 50)        15000       product[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "reshape_1 (Reshape)             (None, 50)           0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "reshape_2 (Reshape)             (None, 50)           0           embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 100)          0           reshape_1[0][0]                  \n",
      "                                                                 reshape_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 100)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 50)           5050        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, 50)           0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 50)           0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 1)            51          dropout_2[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 1,450,401\n",
      "Trainable params: 1,450,401\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = DeepCollaborativeFiltering(n_customers, n_products, n_factors)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "WARNING:tensorflow:From /home/jovyan/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "Train on 100188 samples, validate on 33397 samples\n",
      "Epoch 1/3\n",
      "100188/100188 [==============================] - 14s 142us/step - loss: 0.0105 - val_loss: 0.0184\n",
      "Epoch 2/3\n",
      "100188/100188 [==============================] - 14s 137us/step - loss: 0.0091 - val_loss: 0.0187\n",
      "Epoch 3/3\n",
      "100188/100188 [==============================] - 14s 139us/step - loss: 0.0078 - val_loss: 0.0193\n",
      "Done training!\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer = 'adam', loss = mean_squared_logarithmic_error)\n",
    "model.fit(x = [customer_idxs, product_idxs], y = ratings, batch_size = bs, epochs = epochs, validation_split = val_per)\n",
    "print('Done training!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Log model and metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "logmodel = exec.log_output(\n",
    "    metadata.Model(\n",
    "            name=\"DeepCollaborativeFiltering\",\n",
    "            description=\"Model for product recommender\",\n",
    "            uri=\"\",\n",
    "            model_type=\"neural network\",\n",
    "            version=execTime,\n",
    "            training_framework={\n",
    "                \"name\": \"tensorflow\",\n",
    "                \"version\": \"v1.14\"\n",
    "            },\n",
    "            hyperparameters={\n",
    "                \"batch_size\" : 64,\n",
    "                \"validation_split\" : 0.25,\n",
    "                \"layers\": [n_customers, n_products, n_factors],\n",
    "                \"epochs\" : 3\n",
    "            }))\n",
    "metrics = exec.log_output(\n",
    "    metadata.Metrics(\n",
    "            name=\"Model for product recommender evaluation\",\n",
    "            description=\"Validating of the recommender model\",\n",
    "            uri=\"\",\n",
    "            version=execTime,\n",
    "            data_set_id=data_set.id,\n",
    "            model_id=logmodel.id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 Get current output directory for model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exporting trained model to s3://models/recommender/1/\n"
     ]
    }
   ],
   "source": [
    "directorystream = minioClient.get_object('data', 'recommender/directory.txt')\n",
    "directory = \"\"\n",
    "for d in directorystream.stream(32*1024):\n",
    "    directory += d.decode('utf-8')\n",
    "arg_version = \"1\"    \n",
    "export_path = 's3://models/' + directory + '/' + arg_version + '/'\n",
    "print ('Exporting trained model to', export_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Export models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-22-58b1f5cc64c6>:2: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
      "tensor_info_users user:0\n",
      "tensor_info_products product:0\n",
      "tensor_info_pred dense_2/BiasAdd:0\n"
     ]
    }
   ],
   "source": [
    "# inputs/outputs\n",
    "tensor_info_users = tf.saved_model.utils.build_tensor_info(model.input[0])\n",
    "tensor_info_products = tf.saved_model.utils.build_tensor_info(model.input[1])\n",
    "tensor_info_pred = tf.saved_model.utils.build_tensor_info(model.output)\n",
    "\n",
    "print (\"tensor_info_users\", tensor_info_users.name)\n",
    "print (\"tensor_info_products\", tensor_info_products.name)\n",
    "print (\"tensor_info_pred\", tensor_info_pred.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-23-da3077ef7d92>:14: calling SavedModelBuilder.add_meta_graph_and_variables (from tensorflow.python.saved_model.builder_impl) with legacy_init_op is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Pass your op to the equivalent parameter main_op instead.\n",
      "INFO:tensorflow:No assets to save.\n",
      "INFO:tensorflow:No assets to write.\n",
      "INFO:tensorflow:SavedModel written to: s3://models/recommender/1/saved_model.pb\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "b's3://models/recommender/1/saved_model.pb'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# signature\n",
    "prediction_signature = (tf.saved_model.signature_def_utils.build_signature_def(\n",
    "        inputs={\"users\": tensor_info_users, \"products\": tensor_info_products},\n",
    "        outputs={\"predictions\": tensor_info_pred},\n",
    "        method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME))\n",
    "# export\n",
    "legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')\n",
    "builder = tf.saved_model.builder.SavedModelBuilder(export_path)\n",
    "builder.add_meta_graph_and_variables(\n",
    "      sess, [tf.saved_model.tag_constants.SERVING],\n",
    "      signature_def_map={\n",
    "           tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: prediction_signature,\n",
    "      },\n",
    "      legacy_init_op=legacy_init_op)\n",
    "builder.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 Restarting of the model serving server\n",
    "\n",
    "In order for a new model to take effect it is also necessary to restart a model server.\n",
    "The issue here is that we are not changing the model version version and as a result, \n",
    "the model will not be updated. To ensure model update, we are here restarting a server -\n",
    "simply killing the running instance, and as a server is installed using deployment, the instance\n",
    "will be recreated. Additionally for pods operations to work correctly from the notebook,\n",
    "it is necessary to create permissions allowing for access to pods in another namespace. \n",
    "Look at the podaccessroles.yaml for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod prefix  recommendermodelserver-\n",
      "pod namespace  kubeflow\n"
     ]
    }
   ],
   "source": [
    "recommender = \"recommendermodelserver-\"\n",
    "if directory == \"recommender1\":\n",
    "    recommender = \"recommendermodelserver1-\"\n",
    "print(\"pod prefix \", recommender) \n",
    "\n",
    "namespace = \"kubeflow\"\n",
    "print(\"pod namespace \", namespace) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current pod name  recommendermodelserver-6d5d5c654-snl99\n"
     ]
    }
   ],
   "source": [
    "# Get full pod name for the current model\n",
    "\n",
    "k8s_config.load_incluster_config()\n",
    "\n",
    "v1 = k8s_client.CoreV1Api()\n",
    "\n",
    "pod_list = v1.list_namespaced_pod(namespace)\n",
    "pod = [item.metadata.name for item in pod_list.items if recommender in item.metadata.name][0]\n",
    "print(\"Current pod name \", pod)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done deleting\n"
     ]
    }
   ],
   "source": [
    "# Delete pod, so that it gets recreated\n",
    "v1.delete_namespaced_pod(pod, namespace, grace_period_seconds=0)\n",
    "\n",
    "print(\"Done deleting\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New pod name  recommendermodelserver-6d5d5c654-xvxf7\n"
     ]
    }
   ],
   "source": [
    "# Verify that the new instance was created\n",
    "time.sleep(20)\n",
    "pod_list = v1.list_namespaced_pod(namespace)\n",
    "pod = [item.metadata.name for item in pod_list.items if recommender in item.metadata.name][0]\n",
    "print(\"New pod name \", pod)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
